Hi @DaKings ,

I was wondering if you have found anything in your tests? 
 

Thanks, 

T. 

  @ThiagoGuedes I have an identical setup to you, same issues.

For now I am using one of the 1G ports instead of the 2.5G port on my modem so I can at least get up to 1G on all ports which isn't ideal. 

As an additional piece of information, I am more inclined to believe the issue is due to flow control on the router's side as I am able to get expected speeds from my ISP (~1.5Gbps) with the other SFP port wired to a 10G NIC.  In the same configuration, I cannot get over 400Mbps on any of the other LAN ports.  I'd expect if the transceiver I was using was that abysmal I would see similar results in both situations.

Tried messing with bandwidth limiting and flow control in standalone and couldn't get any better results.

If the issue is on the router side, I hope this can be fixed by a firmware update, or I will not be purchasing a piece tplink equipment again.

 Hi @thePrince, 

Yesterday I had a remote sessions with someone from TPLink. And... news aren't that good. Apparently, according to the support guy I spoke to, this is not compatible. I couldn't really understand the reason, it made no sense to me, but it seems that because the router can understand only 1 or 10g at the optical side, whatever different speed you negotiate at the other side will drop packets. But again, that's what I got from the chat, I could be wrong. 

I've explained that the use case is exactly the reason why these modules were designed for - to connect SFP+ ports to RJ45 in other speeds but no dice. 

He promised to take this back to R&D to check,  but I honestly don't know what to say at this point. 

I'd suggest everyone with the same issue opens a ticket so hopefully they can see reason. It made no sense to me why this won't work. It's almost like saying the SFP+ ports are only to be used optical to optical. Then, why would the sell transceivers? 

Hi  @thePrince ,

For whatever reason, I got a new reply from you that's now showing here. But yes, I totally agree with your points. 

For reference, I'm using TPLink's own transceivers - TL-SM5310. But I tried with another MikroTik before and the results were the same so I'm also inclined to believe there's something wrong with the router. 

What's making me really upset is TPLink trying to say that my use case is not possible or not supported. This is the last reply I got: 

Dear customer,

Thanks for your cooperation.

After confirming with our R&D colleagues, the 5310T itself cannot directly use the SFP port, nor can it be negotiated by changing the 10000M of the SFP+ port to 1000M. This is determined by the nature of the 5310T itself. Other manufacturers, such as UBNT, etc. actually the same as us. 

To sum up, for your problem, our suggestion is to continue to use the electrical port, because your current network demand is within 1G. In addition, if you insist on using the 2.5G electrical port, in order to avoid packet loss when your network is at its peak, we suggest that you connect the client to different LAN ports of the router.

  @ThiagoGuedes 

Yeah I ended up deleting it cause it was really only a valid point (I blame a long day at work for the rant) with regards to the upload side (Router > Transceiver) but your problem is download speeds.  

Going from 2.5Gbps to 10Gbps domain shouldn't really have any issues when it comes to buffering and frame drops on paper, at least from the transceiver side, so the issue we are having is something different entirely than what I explained in my since deleted reply.  Suspiciously I am seeing download speeds cap to 0.4Gbps (1/2.5=0.4... very suspicious), but like I said only on the 1G lan ports, the 10G port gets full speed.  I'll be looking into this further over the next week, I have a few possible leads.

If you were seeing this issue with your upload speeds from the Router to Transceiver, that could totally be because either your transceiver is junk not sending pause frames or TP Link's flow control is.   

TP link doesn't say they support transceivers that do 2.5/5G, because the SFP interface can't be clocked at anything other than 1G/10G.  So support isn't saying anything misleading or technically wrong, this is common for pretty much any SFP+ device I am aware of.  My concern again is just that if the issue ends up being flow control or some internal function, and not any clock domain crossing issues, TP Link should fix that.  It's on us to prove that though, because our use case though not supported, may show a bug in a supported feature... If that makes sense.

  @ThiagoGuedes 

Thinking about it a bit, I doubt TP-Link is to blame here.  I am almost confident it is the modem provided by your ISP (Rogers).  You likely have an XB7 or XB8, same as myself.  I am almost confident the issue is that it does not have any sort of flow control.

This is why I am seeing the speeds are fine for my >2.5G clients on my router hooked up via the other SFP+ port, since its impossible for it to be overwhelmed by the slower 2.5G link. 

The 1G clients however can easily be overwhelmed if the modem is relentlessly cramming frames through the 2.5G link, ignoring pause frames and cramming data that will eventually be lost by the receiver downstream.

I am going to try this out later, its kind of hacky and will make use of the ER8411's ability to multi-WAN, but it should work.  Make sure your modem is out of bridge mode so you can use multiple ports, yes you will have double NAT for now, but so what.  Make 2 WAN connections to your modem, direct clients capable of >=2.5G to the 2.5G link for WAN, connect all others to the 1G link using static routes.  If I am right this should workaround what I suspect to be the issue, and all clients will have their max speed to WAN.

I'll try this later, and if it works I'll just have to figure out how to get rid of the double NAT issue...

Hi @thePrince,

Thanks, that explains why I could not see your earlier reply. 

I was thinking about what you said, but if the 1G clients were being saturaded, enabling Flow/Bandwidith control over the Switch interfaces should remediate the problem. I've tested with either or just one of them enabled and it made no difference. 

You're also right, I'm using an XB7 from Rogers. In addition to the double nat problem, you wouldn't really be getting the 2.5 uplink to the 8411 so that kinda defeats the purpose of having it in the first place. One thing that I thought was using a media converter with 1 mult gig and 1 SFP+, then using a simple DAC between it and the router. But I think that would add even more complexity to the equation. 

To be honest, I don't think the problem is in the SFP+ port or the module for that matter. For me, it's in how the router is handling the Flow Control at the switch level. 

I replied to TPLink's team, let's see what they come back with. 

Best,

T. 

  @ThiagoGuedes 

Switch level is fine for me.  On the LAN my 1G clients can communicate with my 10G clients at full speed. So I don't think that's the issue.

  @thePrince 

I'm far from a network engineer, I just work with IT so maybe I'm wrong. But let me try to explain it in a different way. 

You theorized that maybe the XB7 is craming 2.5 speeds through 1G clients which are dropping packets as result. Here's why it doesn't make sense to me. You can enable Flow Control both at the WAN Level and Switch Level. You can also enable Rate/Bandwidth control at the switch level. 

I don't know how the 8411 is wired internally between WAN and LAN, but if what you explained was the case, enabling Flow Control and/or Bandwith control at the Switch ports would remediate this issue. And this does't happen. 

Putting in a different way: 

Imagine instead of the transceivers, you have just a 10G fiber connection. Obviously, no 1G device can get that speed, but the reason it exists is to distribute that bandwidth to the 8 remaining lan ports, so they could in theory get 1G speed each at the same time and the 10G link would not saturate. This should be the same case we are talking in here. 

In fact, it is: From 8411 perspective, the transceiver acts as 10G connection and the conversion happens at the transceiver level. 

I thought about the XB7 not respecting PAUSE frames. But again, I can be completely wrong and happy to be corrected, but it shouldn't matter much, because the 8411 should be able to distribute up to a 10G link to the LAN. A LAN Switch port shouldn't have trouble signalling to WAN controller that it can internally work at 1G at most. 

Besides, 802.3x has been around for sooo long, I'd highly doubt it's not supported. 

Am I to off here? 

Best,

  @ThiagoGuedes 

Not a network engineer either, I come from a background of embedded systems.

As I said earlier, at the L2 level the routers switch is fine communicating between 10G and 1G clients, so that part is at least fine.

With regards to the WAN connection, I would argue where the router might have trouble is what to do if the ingress frame rate from the modem simply exceeds what the destination can handle.  How does it decide what frames to drop and which to send other than just forwarding whatever it gets to the port, and dropping them if the buffer is full.  Flow control only works if you can prevent the source of the traffic from exceeding the slowest link rate in the chain too, otherwise somewhere will eventually drop the packets.

However, TCP should handle this at L3 which is likely where the problems start here because this is going from WAN to LAN.  The router has to receive the entire packet prior to forwarding it to the LAN.  If it receives the packets faster than it can retransmit them, eventually some buffer will fill up, and they'll be dropped.  That's why flow control is kinda looked down on, it can't solve this problem.  Flow control is supposed to prevent overflow on small time scales.  For example, I receive a packet at XGbps, and retransmit it frame by frame at <=1Gbps to be sure not to overflow the ingress buffer of the destination interface.  If the router isn't doing this, it's at fault.  If the modem for whatever reason is overwhelming the router itself, there's nothing it can do, frames will be dropped, packets will eventually be lost.

At L3 TCP should determine packets aren't getting acknowledged and drop the speed down, so it's not an end-end L3 problem, it's definitely modem to router that is the problem.

So I see it now as there are two possible issues:

1) er8411 is in fact at fault, it's overwhelming the ingress buffer of the 1G destination interface and frames are being dropped (flow control is broken)

2) modem is not respecting PAUSE frames, overwhelming ingress buffer of the router itself, therefore dropping frames and packets.

I see 2 as the more likely scenario.