What is it with all these random questions?

Why aren't you participating in any of the conversation you started?

Rebel, anti-authorian, no-future, good-for-nothing-except-kicking-nazi-ass 'youth' (who reach their 60s nowadays).

Du coup, ça donne plutôt l'impression que l'IFOP est complaisante vis-à-vis des étiquettes que les différents partis s'auto-attribuent à un peu toutes les positions du spectre?

Je dois avouer que je ne sais pas trop comment EELV s'autopositionne.

Alors chez les LFI et les écolos (pas trop EELV il est vrai) que je connais "soc-dem" est plutôt une insulte... Y a des gens à gauche du PS qui se revendiquent de ce terme?

Après oui, "droite radicale" pour un parti de nazillons, c'est abusé.

Rezin fournit de l'accès net en Isère, il paraît qu'ils sont sympa.

Sadly true

Quite frankly, in the last two years, I've accepted the AI term because that's pretty close to what we expected sci-fi AI to be.

Something that you talk to that understands from the context what you mean and that calls the appropriate tool is exactly things like Jarvis, like HAL 9000.

AI can be fine, I cringed at the mention of Starlink, even though that's a case where it makes sense.

Ha ha no, I never went as far as needing embeddings for a language model. MNIST is actually, you know, the very simple classification model. It's a bit the 'Hello World' of machine learning. It's a dataset of handwritten digits that you have to classify in 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

It is a good test because you can train it in minutes if not seconds even on crappy hardware and unoptimized code.

So when I'm talking about negative numbers, I am talking about negative numbers. I am talking about weights that are needed to be negative to have negative influence on the output. Like "this pixel in the center is white, so the likelihood of the number being zero decreases."

I still don’t understand gradient descent fully, but can you explain why you think it should be replaced and with what?

Honestly, I'm just talking about it because we are being silly, but I am not sure that's an idea I actually want to defend.

I just have this feeling that gradient descent is a good mathematical construction for what we try to achieve, but that mathematical purity maybe, just maybe, gets in the way of efficient computing. Of course, there are thousands of very competent, highly paid people who already explored that venue, so I'm pretty sure that if something better was possible and within the reach of one person, it would already have been discovered.

(Counterpoint: we routinely rediscover things that were invented in the 90s that are now good ideas now that we have very good computing)

The thing is gradient descent is used to tell you in which direction you're supposed to move a weight to lower the loss of your results. In other words, to minimize the error of your network.

Gradient or partial derivatives are like an ideal mathematical tool to do that. We are able to derive it for a lot of functions, linear or not, and it is a well-studied mathematical object, so it really makes sense to use that.

The direction of the gradient will tell you the direction in which the parameters need to move. More precisely, the partial derivative of a given parameter will tell you if you need to increase it or lower it in order for the loss to improve.

Thing is we use the sign that's clear but the intensity I am not sure it is that relevant because we keep fighting against things like gradient vanishing problem where very deep networks tend to have very low gradients and we compensate a lot of its problem through optimizers, choices and tricks.

I wonder if there would not be a pure computer science way of just keeping track of the direction in which you want the parameter to change.

I don't know, maybe triple all the calculations by one tick in both directions? or just use gradients on one bit when it makes sense? Or find a function that's very fast to compute but that just approximate gradients and that is just better than randomness at finding the sign.

Like I said, that's just an itch to scratch. That's not a strong conviction that there is something. But if you were to give me two weeks salary to just work on that, I would be very happy to.

At one point I had a weird obsession in making neural networks train only on uint_8. I tried:

0 to 255 is all you get. You know, "Real Programmers scorn floating point arithmetic."

You want 16 bits? Make it a 8 bits overflow counter.

We don't need divisions or multiplications when we have bit shifts.

My end goal was language models (probably not "large") but I barely got to make an acceptable MNIST after begrudgingly accepting that I should sully my 8 bits purity with negative numbers.

I still have that itch to scratch that I feel the process of gradient descent could be replaced by something better designed for the type of information we want to flow back ("move that thingie in that direction for the loss to go down")

Basically I liked the idea of easy visualization and forcing myself to not use any sort of layer normalization (that I secretly suspect I never fully understood)