Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)T

th3raid0r

@ th3raid0r @programming.dev

Posts
8
Comments
18
Joined
2 yr. ago

  • Featured

    April Discussion: What problems are you working on.

    Jump

  • I'll go first - I'm working on the reliability of some migration patterns between a couple managed cloud implementations. A bring-your-own-account model underpinned the legacy setup, while the newer cloud offering allows us to retain it all in our accounts.

    This is on top of generally sussing out the reliability of the newer product with some chaos testing.

    The unfortunate bit is that my OPS workload is so high that I struggle to get much traction on the above. Not to mention that this new product moves so fast that it's hard to get any sort of week-to-week consistency. Often requiring environment re-provisioning. Not exactly stuff that the community can really help with, but hey, that's what's on my plate.

  • SREsyphus

    Jump

  • Definitely not that lucky. We have customers who seem to watch dashboards and create a Sev1 anytime latency degrades by 10%. They explain to their account manager that they need to have perfect performance all the time. The AM then comes to us demanding that we increase the sensitivity of the alert. Management agrees. And then, voila, just like that we have an alert that flaps all day and all night that we aren't "allowed" to remove until someone can show that the noise is literally stopping us from catching other stuff.

    It's insanity.

    EDIT: I only stay because new leadership seems like they want to fix it earnestly. And things are headed in the right direction, but it takes a long time to turn a ship.

  • Anyone else's company getting hounded by Anaconda.com folks?

    Jump

  • The real kicker is that I'm fairly sure we aren't really using them at any real scale - if we do it's to demo our product within the context of AI development. So if anything, they get a lot of free press when we do that. If they're gonna throw a fit over it, I'm sure we can work with some other "AI" company (that's what they bill themselves as) that wants the free marketing. Heck, I can't imagine the anaconda ecosystem working out if they keep threatening the developers that enrich that ecosystem.

  • Anyone else's company getting hounded by Anaconda.com folks?

    Jump

  • Yes, I am aware.

    I'm more asking if others are getting a wide spread of threatening messages across the org - even if they don't regularly use conda/anaconda.

    It's like everyone glossed over "I don't use it in my job at all, and neither do my teammates" bit.

  • Anyone else's company getting hounded by Anaconda.com folks?

    Jump

  • Send the emails to your company’s legal team. It’s not your fight.

    Already did, and agreed. I also asked the legal team if they could ask Anaconda.com to stop contacting me and threatening me personally. We shall see what happens.

  • Anyone else's company getting hounded by Anaconda.com folks?

    Jump

  • I'm not here to discuss the nuances of a startup versus medium sized company. Suffice it to say that much of the organization still views itself as a startup. Even though yes, you are right, it's a medium sized organization.

  • Anyone else's company getting hounded by Anaconda.com folks?

    Jump

  • Sure. I can agree that my company would be liable. But the company isn't mine. I just work here. And my team doesn't use any conda stuff at all.

    Essentially, I am being personally threatened of a lawsuit even though I have no ability to make a licensing or purchase decision.

    That just doesn't sit right with me.

  • Programming @programming.dev
    th3raid0r @programming.dev

    Anyone else's company getting hounded by Anaconda.com folks?

  • Deleted

    Permanently Deleted

    Jump

  • TPMs can be extracted with physical access

    Sure, but IIRC, they'd still need my PIN (for TPM+PIN through cryptenroll). I don't think it's possible to do TPM backed encryption without a PIN on Linux.

    EDIT: Oh wait, you can... Why anyone would is beyond me though.

  • Deleted

    Permanently Deleted

    Jump

  • This sounds like a lenovo machine. Or something with a similar MOK enrollment process.

    I forget the exact process, but I recall needing to reset the secureboot keys in "install mode" or something, then it would allow me to perform the MOK enrollment. If secureboot is greyed out in the BIOS it is never linux's fault. That's a manufacturer issue.

    Apparently, some models of Lenovo don't even enable MOK enrolment and lock it down entirely. Meaning that you'd need to sign with Microsofts keys, not your own. The only way to do this is to be a high-up microsoft employee OR use a pre-provided SHIM from the distribution.

    https://wiki.archlinux.org/title/Unified_Extensible_Firmware_Interface/Secure_Boot#Using_a_signed_boot_loader

    For that case, Ubuntu and Fedora are better because, per the Ubuntu documentation they do this by default.

    On Ubuntu, all pre-built binaries intended to be loaded as part of the boot process, with the exception of the initrd image, are signed by Canonical's UEFI certificate, which itself is implicitly trusted by being embedded in the shim loader, itself signed by Microsoft.

    Once you have secureboot working on Ubuntu or Fedora, you could likely follow these steps to enable TPM+PIN - https://wiki.archlinux.org/title/Systemd-cryptenroll#Trusted_Platform_Module

    There might be some differences as far as kernel module loading and ensuring you're using the right tooling for your distro, but most importantly, the bones of the process are the same.

    OH! And if you aren't getting the secureboot option in the installer UI, that could be due to booting the install media in "legacy" or "MBR" mode. Gotta ensure it's in UEFI mode.

    EDIT: One more important bit, you'll need to be using the latest nvidia drivers with the nvidia-open modules. Otherwise you'll need to additionally sign your driver blobs and taint your kernel. Nvidia-Open is finally "default" as of the latest driver, but this might differ on a per-distro basis.

  • Deleted

    Permanently Deleted

    Jump

  • Yeah, no kidding. The same systemd that enables the very things OP is trying to enable...

    systemdboot + sbctl + systemd-cryptenroll and voila. TPM backed disk encryption with a PIN or FIDO2 token.

    AFAIK this should be doable in Ubuntu, it just requires some command-line-fu.

    Last I heard the Fedora installer was aiming to better support this type of thing - not so sure about Ubuntu.

  • Booting Linux off of Google Drive

    Jump

  • Hahah, good luck. Proton Drive is really terrible. I can't even upload a single 1GB file through the service.

  • Booting Linux off of Google Drive

    Jump

  • Well, I mean, most corps trying to shoehorn AI into things are using Cloud implementations of the various "AI" solutions.

    What, pay for our own datacenter? Nah.

    Just import openai and add "the AI" that way. 🤦‍♂️

  • Linux @programming.dev
    th3raid0r @programming.dev

    Booting Linux off of Google Drive

    ersei.net /en/blog/fuse-root

  • Icebreaker Discussion - Dealing with ableism - What works and what doesn't?

    Jump

  • Fair, and I think I'd have gone that direction if it wasn't a slack channel where everyone was invited to, and then questioned if they decided to leave. It was also a very noisy channel where it was disrupting my work.

    I didn't just throw this into some channel in which I wasn't invited or anything. I actually felt like I wasn't allowed to leave, which is why other NDs privately thanked me afterwards.

    I can ignore the ignorable, but if you're going to hunt me down if I ignore it (like they were doing), then I needed to speak up in order for it to stop.

    Typically I do just what you've described, just kinda ignore it.

  • Autistic Devs @programming.dev
    th3raid0r @programming.dev

    Icebreaker Discussion - Dealing with ableism - What works and what doesn't?

  • Autistic Devs @programming.dev
    th3raid0r @programming.dev

    Soliciting Art for this Community

  • Programming.dev Meta @programming.dev
    th3raid0r @programming.dev

    I created autistic_devs community to provide a safe(er) space to discuss our unique issues within the field.

    programming.dev /c/autistic_devs

  • Is it possible to "manage up" on customer expectations? Or am I doomed to unreasonable SLAs? (Database as a Service Company)

    Jump

  • It's not even a steaming pile of crap or anything. Since it's basically a managed distributed database solution there's limits to what we can do and maintain strong consistency. Things generally take a long time and are very sequentially dependent. So we have automation of course! Buuuut there's very little comfort or trust in what is now very well exercised automation - which is the number 1 barrier in removing many sources of toil. Too many human "check this thing visually before proceeding" steps blocking an otherwise well automated process.

    We are so damn close, but some key stakeholders keep wanting just one more thing in our platform support (We need ARM support, We need customer managed pki support, etc.) and we just don't get the latitude we need to actually make things reliable. It's like we're Cloud Platform / DevOps / QA / and SRE rolled into one and they can't seem to make up their damn mind on which rubric they decide to grade us on.

    Hell they keep asking us to cut back our testing environment costs but demand new platform features tested at scale. We could solve it with a set of automated and standardized QA environments, but it's almost impossible to get that type of work prioritized.

    My direct manager is actually pretty great, but found herself completely powerless after a recent reorg that changed the director she reports to. So all the organizational progress we made was completely reset and we're back to square one of having to explain what we want - except now we're having "kubernetes!" shouted at us while we try to chart a path.

    I'm already brushing up my resume, but I must say, the new Gen-AI dominated hiring landscape is weird and bad. Until then, I just have to do the best I can with this business politics hell.

  • Is it possible to "manage up" on customer expectations? Or am I doomed to unreasonable SLAs? (Database as a Service Company)

    Jump

    1. They want to be notified of anything that could potentially slow down their system. So any anomaly. The catch being is that they constantly change patterns because they introduce new workloads weekly - which wouldn't be a problem if they could better communicate their forecasts. And that's just one of a few dozen customers - again all with unique cluster configuration and needs.

    2. Yeah, it sucks. The first year was pretty great and we had a fully integrated and unified managed services team where we were getting some great automation done. Then they split the team in half in order to focus on a different flavor of our product (with an entirely new backend) and left folks who were newer (myself included) with maintaining the old product. We were even told that we should be doing minimal maintenance on the thing as the new product would be the new norm. Then once upper management remembered how contracts work, they decided we needed to support 3 new platforms without growing the team. All while onboarding new customers and growing the environment count. We're now in operational overload after some turnover that was backfilled with offshore support that has a very minimal presence.

    3. I have tried championing this, but I don't expect an ableist, masculinity shaming person like you to understand a call for social pointers on how to "manage up".

    "Man Up" - good lord, way to be an ass.

  • Is it possible to "manage up" on customer expectations? Or am I doomed to unreasonable SLAs? (Database as a Service Company)

    Jump

  • I’d suggest to just set up automatons to fix those things automatically. Lets say 80% CPU for 5 minutes it too high. Ok, add an auto-scale rule at 65% CPU for 3 minutes to add an extra node to the cluster to load balance the CPU load

    Sure, if it were a normal service and not a distributed database that requires days to scale. Days. It's not, "add one node" and we're good. It's Add Node - Migrate Data - Add Node - Migrate Data... And in many cases, we have explicit instructions NOT to scale the customer because they won't be able to afford the larger cluster.

    Also, would you auto-scale for a 5 minute blip that goes away in that time and doesn't consistently recur? I certainly wouldn't. The customer might not be able to pay for the size we put them on.

    Our customers can simultaneously demand that we respond to all alerts AND not to scale their cluster. Who's fuckin' idea this was, I've no clue.

    Like it sounds like you’re saying the issues are caused by systems not being robust and lack of automation… If they’re this scared of outages and breaking SLA, they should work on having less outages, or having fall-backs when they occur.

    No. That's reading far more into my statement than I hoped. The reliability is indeed there - it's VERY unlikely our managed database goes down to a technology issue in our control. If it does, it's usually an operator error thing. However, if it were down to just operator error alerts and things actually impacting the end users, my job would be a dream!

    Automation is somewhat there, but there's a few stakeholders that insist on human validated steps. So, while I have an ansible playbook for most issues, operating that playbook takes hours.

    But it could get pretty difficult to get management to do this kinda things from random suggestions from some SRE. I’d probably talk with the team-lead about this, and other people in your team, cause you’re probably not the only one with these issues. And then have a meeting with the entire dev/SRE team and management to point out it’s not sustainable the way it’s going, and with suggestions to improve it

    Sure, if it were technical. But this is largely not a technical issue, as you had assumed. The issue is that there is someone, with power, who gets to say that we must follow unreasonable customer requests to the letter. Even if those requests run counter to our sustainability.

  • Is it possible to "manage up" on customer expectations? Or am I doomed to unreasonable SLAs? (Database as a Service Company)

    Jump

  • In the most recent case, perhaps that could be a fix. Sure. But that won't work for every customer we have. Its certainly an idea worth bringing up to the team provided management doesn't shoot it down. Thanks!

  • Ask Experienced Devs @programming.dev
    th3raid0r @programming.dev

    Is it possible to "manage up" on customer expectations? Or am I doomed to unreasonable SLAs? (Database as a Service Company)