Mysteries of Telegram Data Centers

(dev.moe)

151 points | by theanonymousone 4 hours ago

16 comments

  • dubbel 1 hour ago
    The article is from May 2022, just fyi.
  • glaslong 1 hour ago
    The DC3 gap is interesting. I wonder whether they deprecated it because the other EU server had plenty of capacity, or still keep it but only for... "special" account data flow.

    Also, it looks like it's easy enough to ID your DC on their API, though I haven't tried it yet (more of a Matrix Stan personally): https://core.telegram.org/method/help.getConfig

  • flexagoon 58 minutes ago
    Just like DC5 is often down to the discontent of Chinese users, DC2 is the one serving all Russian and Ukrainian users, so in the more technical Russian-speaking communities "dc2 down" is also a pretty common saying
  • SpaceL10n 1 hour ago
    For a hot second there I was really excited to learn about historical telegram "data centers".

    It's a capital T.

  • strebz 2 hours ago
    DC2 is the first connection point of all MTProto clients.

    Any DC may refuse a request and force the client to switch DC.

    Profile URL doesn't show where messages/chats/channels are stored, as telegram has two dedicated DCs mostly for media. The rest DCs allow media with bandwidth being throttled.

  • AntronX 2 hours ago
    DC in Miami, explains why Telegram app is snappy fast for me. I notice similar speed improvement with Meta and other big tech apps when I'm on the west coast. I guess latency matters when your app is making tons of requests.
    • Anon1096 2 hours ago
      Most of big tech's major data centers are in Loudoun County, VA on the east coast not the west coast. It's centrally located to be great latency for the east coast and OK latency for west coast and Europe. Plus a friendly regulatory environment and lots of existing DCs (AWS us-east-1, Azure East US 1/2)

      If you're feeling any better latency on the west coast it's more likely to be placebo than real.

      • tobinfekkes 2 hours ago
        On the contrary, Big tech famously has plenty of data centers on the west coast:

        • Quincy, WA (Microsoft)

        • The Dalles, Oregon (Google)

        • Prineville, Oregon (Facebook, Apple)

        • Hillsboro, Oregon (Cloudflare, others)

        • Boardman, Oregon (AWS)

        • Anon1096 1 hour ago
          By traffic load east coast datacenters dwarfs these.
          • tobinfekkes 1 hour ago
            The parent comment wasn't regarding traffic loads; it was regarding latency.
            • Anon1096 15 minutes ago
              Your experienced latency will be heavily influenced by which DC you hit, in which case we care about the relative traffic loads because you aren't just hitting your closest one all the time.
  • _ink_ 2 hours ago
    They claim that they store user data on different servers in different jurisdictions so it becomes more difficult for authorities to gain access [1]. Maybe that's true and it has something to do with these DCs that seem to be unused.

    [1] - https://telegram.org/privacy

    • amima 2 hours ago
      They do not claim that. They do claim that they store specifically encryption keys in several data centers in different jurisdictions. Here is the exact quote: "All data is stored heavily encrypted and the encryption keys in each case are stored in several other data centers in different jurisdictions". So only keys are distributed.
      • hkpack 1 hour ago
        What does heavily encrypted even mean? Fully encrypted? Slightly encrypted? Encrypted enough to call it “heavily encrypted” but not enough to be protected from whoever is interested?
        • Perz1val 42 minutes ago
          Heavily means the key is large so it takes longer to crack, but also longer to encrypt/decrypt, so the service is more costly to run and slower. At least I've seen it used that way
          • maqp 32 minutes ago
            There's nothing slow about AES.

            In this context "heavily" means "we can't legally claim it's end-to-end encrypted because it's not".

            Also it's not even post quantum, so it's not heavy. Telegram's Diffie-Hellman breaks instantly with a quantum computer large enough to run Shor against it.

            Also, the keys sit on the servers' RAM, no matter what they lie. There is no global distributed RAM system, especially one that encrypts data in distributed fashion and works at the negligible latencies that Telegram boasts.

        • dakolli 34 minutes ago
          telegram is the safest encrypted messaging app. Period, full stop.
    • bflesch 1 hour ago
      It's more about the fact that five eyes intelligence services prefer to officially spy on each other's countries so they don't have to answer to their respective bureaucrats. They prefer plausible deniability.

      Something like this:

         DC1 politically belongs to UK which "spies" on CA/US but physical servers are located in US so US ultimately retains control.
      
         DC2 politically belongs to France which "spies" on RUS/UKR/DE but physical servers are located in NL (e.g. in UK because one wouldn't be able to spot difference in ping). Maybe it's politically owned by UK/NZ or UK/AUS because France can't be publicly caught spying on Germany. But France wouldn't risk public arrest of Telegram CEO and the spectacle with russia if there is nothing to gain.
      
         DC4 politically belongs to USA which "spies" on UK/Israel but physical servers are in NL/UK
      
         DC5 politically belongs to UK/USA which "spies" on AUS/China/India but physical servers are in Singapore (e.g. former UK colony)
      
      
      I love mentioning the UK in these kind of discussions because the pushback is biggest every time the Crown is mentioned, and ultimately US/CA/NZ/AUS are all colonies under the King.

      Really cool to see realpolitik mapped out like this. It also highlights the problem of metadata with these kind of topics.

      • orbital-decay 20 minutes ago
        The reason it's in Singapore is that Telegram can't operate in China, and Singapore-washing is the closest thing to doing it. A ton of VPNs and other services targeting mainland users but not allowed in the mainland are hosted there, it's a huge hub for companies and networks.
  • londons_explore 3 hours ago
    This strikes me as a huge amount of custom code and technical debt. Every new software dev probably has to learn this.

    Why not a sticky master election per user, and have no special data centers?

    • codedokode 2 hours ago
      It makes sense: European users are assigned to EU data center, and Chinese to the one closer to them. The "custom code" should not be complicated, just a map of country to DC.

      You are suggesting to develop a compicated solution (spend money) when current simple one is working ok without any elections.

    • fullstop 3 hours ago
      From what I have read, they only have ~30 employees. They're not exactly onboarding a lot of new people here.
    • inigyou 2 hours ago
      If you've ever actually tried to implement server clustering you quickly find there's no magic cloud, except in specific cases like blockchains. A privately operated cluster system is mostly about directing requests to the correct server out of a finite set of servers.
    • nurumaik 2 hours ago
      Learn what? How to count to 5?
  • bflesch 2 hours ago
    Beautiful analysis. It really looks like the country distribution [1] follows the geographical split between five eyes intelligence services, and maybe a small slice for France after they imprisoned the Telegram CEO [2] in order to take over data ownership from russia.

    [1] https://dev.moe/wp-content/uploads/2022/05/image-14.png

    [2] https://en.wikipedia.org/wiki/Arrest_and_indictment_of_Pavel...

  • hocuspocus 2 hours ago
    I'm on DC5 since I lived in Korea when signing up, but I cannot say I've noticed many outages.
  • DOGMATICA 3 hours ago
    i'm far from an authority on content delivery or whatever, but the first thing I thought of was what a bizarre way to setup your infrastructure!
    • Ghoelian 3 hours ago
      idk, they probably tried to get people on DC's as close to their location as possible. Using your phone number's country code might seem like a good way to do this at first, and they probably didn't give it much more thought before building the whole thing on this idea.
    • 486sx33 3 hours ago
      [dead]
  • hashtag-til 3 hours ago
  • hhh 3 hours ago
    something smells suspicious about this kind of data routing
    • overallduka 2 hours ago
      The Lex Fridman podcast episode with Pavel Durov is worth listening to. Their servers are built to be very secure — of course, it would be different for others, and they use some clever tricks
  • dgroshev 2 hours ago
    • overallduka 1 hour ago
      Good story, I yet believe the guy is trying to do the right thing. In the lex Friedman podcast he talks about banning extremist channels in both sides always, the story focus more on the Nalvani's block, but accordingly him he also bans other sides depending on the content. I do follow a number of Telegram channels about the Ukraine war, and the pro Ukraine channels are there together with pro Russian channels.
      • dgroshev 1 hour ago
        Of course he would say that. Even besides the fallaciousness of this argument (there is no equivalence between an aggressor and a victim), Pavel Durov is completely untrustworthy.

        There is product stuff, like misleading claims about Telegram's encryption and comparisons to Signal. In reality, for the vast majority of chats they have plaintext, unlike Signal.

        There are more subtle positioning claims. Durov made a huge deal of his "exile", but I saw Telegram's office in St Petersburg with my own two eyes a year after Durov "fled the country". It definitely wasn't shut. There were even local news of Pavel assaulting a guy in St Petersburg for trying to make a photo of him a couple years later.

        And then there are just completely unnecessary lifestyle claims. He said multiple times how he doesn't take any "pills" or medications. It only takes a minute to find his old photos. Male pattern baldness doesn't stop progressing without DHT suppression, and last time I checked, finasteride comes in pills. I don't understand why would he make misleading statements about something so visible, but it doesn't make him any more trustworthy.

    • vvpan 1 hour ago
      It's a great an telling investigation. Dropped in to share it as well. Telegram deserves no trust from us.