Add support for usetting pod identity with spark-run

[Qmando]

As described in an internal tech spec document. Support for using pod identity for manual spark-runs.

Pairs with Yelp/service_configuration_lib#155

1. Check yelpsoa-configs for IAM roles assigned to a particular service when running spark-run. Use those to create a service account, and pass that to the spark conf builder
2. Allow you to override this, but it must be a valid role for at least one instance of the service
3. When this is used, don't send IAM credentials to the spark conf builder. They are still passed to configure_and_run_docker_container, where they will be used for the driver
4. If no aws role/profile/user is specified, use a default one (mrjob as placeholder. this will change in the near future to the spark-driver-k8s user) for the driver. This user can list files in S3 and write to the spark logs buckets.

Test run using spark integration test, as if iam_role set: https://fluffy.yelpcorp.com/i/mmcV6DNvz0zTbC6VJScBcFjKw3Dd466L.html
Test run with no iam_role set, --aws-credentials-yaml provided instead: https://fluffy.yelpcorp.com/i/WDm3kj0R3sg9rVqMkzMmzlJVpSSRhBNc.html
Test with unauthorized forced pod identity: https://fluffy.yelpcorp.com/i/XQR1VpgjCQXLfbd6CfJbdGkwGlKX4ZS5.html

Deployment plan:
If this gets merged, I'll update package version in dev and verify all the tron-based spark jobs are still succeeding as normal.

[nemacysts]

whoops, meant to add this note to my review:

@Qmando: probably wanna wait for someone on ml-compute to ship this as well since they own this, but lgtm :)

[chi-yelp]

1. Check yelpsoa-configs for IAM roles assigned to a particular service when running spark-run. Use those to create a service account, and pass that to the spark conf builder

Just to confirm - we're only getting the service accounts names here for executor pod identity, not creating them right?

[nurdann]

1. Check yelpsoa-configs for IAM roles assigned to a particular service when running spark-run. Use those to create a service account, and pass that to the spark conf builder

Just to confirm - we're only getting the service accounts names here for executor pod identity, not creating them right?

Yes, even currently you can only use SA that already exists on the namespace. This further filters it so that only IAM roles specified in yelpsoa-configs can be used.

[Qmando]

@nemacysts @chi-yelp
I've put all the change in behavior behind a new arg, --executor-pod-identity in the latest commit. I was hoping to avoid having to do this, but I realized there will be cases where this change could break existing jobs. See https://yelp.slack.com/archives/C08920ZF0Q2/p1743024040622489 for more details.

This change makes this PR much safer to deploy, but it also means we'll need to be more thorough with manual testing. I hope to deprecate it in the future when everything is setup correctly.