I've noticed a similar thing. I had many more friends as a small child than I did as a preteen, teen, young adult, or adult.
With small children, so much of their conversation is random and based on interests even an autie/aspie kid can join in with... cartoon shows, video games, animals, dinosaurs, machines, made-up stories. As a kid, I had a group of about 5 I played with regularly, and I furnished most of the more plot-based imagination games, which were usually based on Star Wars or pretending to be animals breaking out of the Pound (that one was basically a mirror to my attitude about school in general). I was also good at thinking up new stuff to do, like catching tadpoles in the ravine behind our subdivision, building a tree fort out of trashed and busted-up building site garbage in the same ravine, building a rather large and fully functional dam in the ravine and going swimming in the large pond that resulted when we left the dam up overnight... fun stuff that required the sort of geekery I was good at. This lasted from nursery school to around kindergarten. KG was also a bit of a wash for me because I spent almost half the year in the hospital recovering from bacterial spinal meningitis and didn't really get the chance to make many friends.
Things began to change for the worst between grade 2 and 3, when the NT kids began to really distinguish between cool and not-cool, and more of their conversation and socializing began to revolve around watching the same TV shows, playing with the same toys, repeating the same slang vocabulary, and excelling at sports.
I was a tomboy and I had an abject fear/hate/distaste toward dolls. Dolls creeped me out something terrible (still do), so I mostly played with plush toys, model cars, and art supplies. And of course, all the girls were stuck on Barbies, baby dolls, fashion dolls, Strawberry Shortcake, Jem, and Cabbage Patch dolls. I didn't even like My Little Pony dolls! I had my plush toy frog, my plush toy mice (still have the frog and the mice), my Matchbox and Tonka toys, and my drawing stuff, and I didn't even want to *touch* a Barbie. I didn't watch the cartoons meant for girls (Jem, She-Ra, Strawberry Shortcake, etc), but I did watch the ones meant for boys (Thundercats, Transformers, Star Wars, He-Man, etc.) So I had no common popular-culture interests with girls. And at that age, boys still think anything female has cooties, even when said female has the same interests they have. So the boys would only play with me when they were sure no one else was around who could see them and then tease them for playing with a girl. I was also hopeless at sports due to vision problems, dyspraxia, neurogenic low muscle tone, left-side weakness that was my only lasting problem from the meningitis, and the fact that I was always the shortest and scrawniest kid in the class. I was clumsy, fatigued easily, couldn't catch a ball to save my life, hated team sports, and couldn't run fast.
Some girls began to really get into clothes at that age too, and I was on the outs there as well. I hated dressing up (still do), hated dresses and skirts (still do), detested pink and other 'girly' colours (still do), dressed like a boy (still do, since I'm asexual and identify more male than female and always did), and had severe sensory issues that limited my clothing options to soft t-shirts and sweat-shirts, track pants, soft cotton pants, or very loose, oversized jeans. Back then, the trend for clothing was for tight clothes, pinks and pastels, and very girly dresses and skirts, so I was pretty much doomed to eternal geekdom.
Between grades 3 and 5 were the worst of my pre-teen years. In grade 4 I ended up with an abusive, bullying teacher who attacked me and 4 others, encouraged the bullies in the school to target us, scapegoated us for everything, and did everything he could to humiliate us. The five of us stuck together out of survival. But that wasn't the root cause of the social shitstorm... it was more of the same. In grade 4, kids broke into social groups based on interests, perceived social status, clothing styles, and parents' social/financial status. My interests were increasingly out of step with those of the other girls, I dressed like a boy and still liked 'boy shows' and 'toys geared towards boys', still hated dolls and anything 'girly', and lived in a family that was lower-middle-class.
We moved the summer before 5th grade, and I went to a new school. I had a better time socially there because I had no 'history' coming with me. No kids there knew that I had been a social pariah at my old school. While things were better and I did make quite a few friends, I was still vastly out of step with the interests of the other girls, still dressed like a boy, was still seen as eccentric, and was still low on the social totem-pole. Grade 6 was the best for me, since I got a teacher who had a real instinct for kids who weren't typical and was very artistic and creative. She encouraged my drawing and helped me create a little comic strip. This little comic helped bridge a social gap between myself and others as the other kids followed it and were interested in the next instalment to go up on the bulletin board.
Grades 7 to 13 went right back to being terrible. During those years, I went around with a metaphoric target on my ass all the time. In HS, I stopped speaking at school almost entirely and I began to self-inflict injury to cope.
As an adult, I have no IRL friends, only friends from online. Part of this is because I work nights, but the main thing is that I just don't fit in. I'm 35, single-by-choice, sterile-by-choice, asexual, have issues with gender identity, and have unique interests. Most women my age are all about their husbands and kids... I have nothing in common there. Very few 30-something women are into graphic novels, pet rats, obscure documentaries, cheesy horror movies, chasing ghosts, and exploring abandoned buildings. So I do my main socializing online.