Some time around 02:00 on Feb 15th we were hit with a distributed denial of service attack, which completely saturated both of our connections to the outside world at 1.5Gbps aggregate traffic. We are working with our upstream providers to prevent further issues if this should happen again.
Five years ago I put up our first tower in Velarde, after testing the equipment for a few months on my deck. This photo was taken as I finished setting it up.
We've come a long way since then. Fewer 3am outages and more customers. At the heart of this business is my desire for something better than what the local telco had to offer. I take joy in seeing people finally able to stream movies, to video chat with their grandkids, to work from home like I was trying to do before.
To all of our customers, I think about you every day. Downtime pains me emotionally. I want the best service we can afford to provide to all of you. Thank you for your ongoing business. We are doing our level best to give you fast, reliable, and affordable internet service.
Power has been restored and all of REDI Net's devices are back online. Thank you for your patience!
We received word that REDI Net is having power-related connectivity issues at seven locations in Espanola. We are currently operating on backup links in Espanola. All clients in Espanola are affected at this time. As we know more, we will provide further updates.
REDI Net’s fiber techs did finally find the source of the fiber break that caused an outage. It was a single piece of shot from a shotgun, which was fired at the fiber at 12:01 AM on January 1st, probably in celebration of the new year. It required manually inspecting hundreds of feet of the fiber with a bucket truck, which is part of what caused the outage to extend for so long.
We were not satisfied with the response time in repairing this break, and some issues with that process will be discussed at the next REDI Net board meeting.
We are also preparing to upgrade the backup links serving the affected area next week, and already have the equipment to do so, so that such a break will not have this effect on our network in the future.
We are currently working on some software on the back end to issue outage credits to all affected customers, and expect credits to be issued by the end of the week.
Our apologies for the delayed update, the HS tower site was brought back online yesterday afternoon at around 16:30 local time.
Thank you all again for your patience.
We are currently experiencing an outage at the Espanola High School Water Tank with unknown cause. We are actively investigating and will provide updates as they become available.
REDI Net service has been restored to our location as of approximately 13:20 local time. We thank you for your patience through the outage.
REDI Net has not yet located the damaged portion of fiber and has not received the necessary components to resume service to our location. We are currently still operating on our reduced capacity backup link. REDI Net has not provided us with an estimated time for the repair to be completed. We are working to reinforce the backup link early this coming week. Thank you for your patience.
As of 4PM today we received notice from REDI Net that they are searching for a small bullet hole in over 2 miles of fiber optic cable to repair the break. It appears it was shot at midnight on Jan 1. In the mean time they have ordered additional equipment to help work around the issue, which is expected to be install sometime tomorrow, Saturday Jan 5th. Once installed it should restore bandwidth to normal for affected customers.
The fiber repair crew was unable to locate the damaged equipment yesterday and they are continuing their search today.
REDI Net has dispatched a repair crew to fix the damaged fiber optic cable. We expect that service should be restored by the end of the day today assuming the crew is able to quickly determine the location of the damage. Thank you for your patience as REDI Net works to repair their equipment.
Per previous posts, we are waiting on upstream provider REDI Net to repair a fiber break in the fiber optic cable providing connectivity to some of our network.
We were able to make some modifications to our backup links providing service to the affected areas to provide more bandwidth. We are currently up to 30-50% of normal capacity in the affected areas.
We will post more updates and update the phone system status message as soon as we have more information.
We were just notified that the upstream provider (REDI Net) outage affecting some of our customers was due to a fiber break. A fiber splicing contractor has been called by REDI Net to locate and repair the break. Normal service will be restored when they are finished finding and repairing the break. We are unable to improve performance until that is resolved.
They have told us it will not be resolved until Thursday the 3rd at the earliest.
It appears at midnight a switch belonging to our upstream provider REDI Net locked up in some manner. This switch provides fiber connectivity to a large part of our network, and as it is not working we have been running on our own wireless backup links since then. Due to the holiday today, REDI Net did not have access the the room in the building where the switch is located. They will have access tomorrow, January 2nd.
We were already working on improving the capacity of these backup connections to avoid such an issue in the future, we are continuing to do so. We will also be discussing this problem with REDI Net and trying to make sure that they have access to all locations 24/7 in the future. We were not aware until this afternoon that they were unable to access this location during off hours.
It has been a very challenging week for us between the weather and now this issue, but we have been working through the holidays to resolve each issue as it has arisen. Over the next few weeks as weather allows, further changes will be made to help avoid such problems should these conditions occur again. Thank you for your ongoing business.
Happy New Year! We are currently operating at a reduced capacity in the following locations due to an offline piece of network equipment:
We do not currently have an estimated resolution time. Thank you for your patience as we work to restore capacity.
We have three sites down now due to extreme cold and/or solar charging issues, we are loading up to investigate all of them today. We are working to get everything back online as soon as possible.
Beginning at 11PM on 2018-12-27 issues with our connectivity in Espanola started happening, originally we thought related to a problem with our connections from REDI Net in that area. That may be true, but shortly after that widespread power outages began in Espanola. We are waiting for more information on what is causing these issues.
The incorrect forwarding problem has been fixed, all 505-852-51XX numbers including the office and support lines should be functional once again.
We are currently sorting out an issue with our phone system with our upstream T1 provider. Incoming calls to 505-852-51XX, including our office line, are not working.
You can still reach us at 844-544-9499 until this is resolved.