• hellinkilla [comrade/them]@hexbear.net
    link
    fedilink
    English
    arrow-up
    0
    ·
    3 months ago

    LLMs are going to start getting worse once more data is fed into them

    I’ve been hearing this for a while, has it started happening yet?

    And if it did, is there any reason why people couldn’t switch back to an older version?

    • semioticbreakdown [she/her]@hexbear.net
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      3 months ago

      I knew the answer was “Yes” but it took me a fuckin while to find the actual sources again

      https://arxiv.org/pdf/2307.01850 https://www.nature.com/articles/s41586-024-07566-y

      the term is “Model collapse” or “model autophagy disorder” and any generative model is susceptible to it

      as to why it has not happened too much yet: Curated datasets of human generated content with minimal AI content If it does: You could switch to an older version, yes, but to train new models with any new information past a certain point you would need to update the dataset while (ideally) introducing as little AI content as possible, which I think is becoming intractable with the widespread deployment of generative models.