Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)S

shaserlark

@ shaserlark @sh.itjust.works

Posts
2
Comments
71
Joined
1 yr. ago

  • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Thanks, that’s very helpful! Will look into that type of build

  • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • I understand what you’re saying but I’m coming to this community because I like having more input, hear about the experience of others and potentially learn about things I didn’t know about. I wouldn’t ask specifically in this community if I wouldn’t want to optimize my setup as much as I can.

  • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Interesting, is there any kind of model you could run at reasonable speed?

    I guess over time it could amortize but if the usability sucks that may make it not worth it. OTOH really don’t want to send my data to any company.

  • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • I’d honestly be open for that but would an AMD setup not take up a lot of space and consume lots of power / be loud?

    It seems like in terms of price & speed, the Macs suck compared to other options, but if you don’t have a lot of space and don’t want to hear an airplane engine constantly I’m wondering if there are options.

  • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Yeah the VRAM of Mac M series is very attractive for running models at full context length and the memory bandwidth is quite good for token generation compared to the price, power consumption and heat generation of NVidia GPUs.

    Since I’ll have to put this in my kitchen/living room that’d be a big plus but idk how well prompt processing would work if I send over like 80k tokens.

  • Deleted

    Permanently Deleted

    Jump

  • Lmao great find. Academic publishing companies are absolute parasites btw, Libgen did the world a favor

  • Your favorite lesser known docker container?

    Jump

  • So would this work well e.g. with the the *arr stack? Because most of the services wouldn’t even need to run always

  • Deleted

    Permanently Deleted

    Jump

  • Nah mega upload is nice but he’s a piece of shit and always has been.

  • Selfhosting GitLab?

    Jump

  • I'm intrigued! But how does it compare to React which is pretty straight forward? I'm not a frontend dev so what's really great about React is that it works super well with LLMs.

  • Selfhosting GitLab?

    Jump

  • Thanks! Super helpful and I’d love to have the compose and install script. I also looked into the Helm charts but still wondering if I should go down that route or not eventually.

  • Selfhosting GitLab?

    Jump

  • Thanks! What about CPU usage, how many CPUs did you assign to the environment you run the container in?

  • Selfhosting GitLab?

    Jump

  • Selfhosting GitLab?

    Jump

  • Thanks! What resources are you running it on? I'm looking into a VPS that could host it and ChatGPT recommends 4-8 vCPUs and 16 GB Ram, which sounds reasonable. But let's say I'm running it on k8s does that leave any room for e.g. running other services on the same cluster?

  • Selfhosting GitLab?

    Jump

  • Thank you! I'm running a Servarr over Docker Compose, and have managed some Kubernetes clusters in the past (although poorly tbh). Any idea how complicated that is in comparison? Also, do you use their Helm charts?

  • Selfhosting GitLab?

    Jump

  • I would deploy the whole app over k8s Helm charts and I would want to use the CI/CD tools and also do Traefik/Ingress for load balancing and having cloudflare point at it. In the future I might be collaborating with other people so I would want the architecture to be solid.

  • Selfhosting GitLab?

    Jump

  • I would actually want to use it to integrate with k8s. I would deploy the app on Kubernetes and do load balancing + pointing at a Cloudflare domain so I would need the whole thing to be solid. I think I do need a lot of the features, but I don't think I necessarily need to have GitLab if something FOSS could offer the same.

  • Selfhosting GitLab?

    Jump

  • Thanks! May I ask what kind of setup you were running and if there's any feature you might be missing that existed in GitLab but doesn't in Forgejo?

  • Selfhosting GitLab?

    Jump

  • Thanks! This looks actually really interesting. Did you try doing CI/CD with it? In future I would probably collaborate with others who'd be also using my self-hosted Git. What would be critical for me is that I can set it up in a way that once I open a PR that branch automatically gets deployed to a dev Kubernetes environment and when I merge with main that it automatically deploys to staging and only when I release a tag the branch would end up in prod. Also I'd like to do secrets management over the platform. I like that Forgejo would be non-commercial and I would prefer it over GitLab if it can do these things well.

  • What are some controversial privacy topics?

    Jump

  • I‘m referring to them handing over the data to law enforcement of the US and other unknown governments.

    What exactly they hand over I can’t tell you, it might be harmless. In the case that they revealed they used push notifications data to identify a pedophile who was using some encrypted messaging service. I hope he gets what he deserves but for us it means we shouldn’t trust anything that uses Apple‘s or Google‘s push notification servers.

    Yeah I know about Molly etc., but the point is, no one I know is going to degoogle their phone and use that. It would be easier if they’d just use a more private, decentralized app that also doesn’t ask for a phone number ffs.