feat: add config option for skipping arrow ipc read validation

[killzoner]

Which issue does this PR close?

Closes #1189.
Revival of #1216 with added configuration.

Rationale for this change

Being able to make use of optimized Arrow IPC reads

What changes are included in this PR?

• Added arrow-ipc-optimizations feature (default enabled) to enable option
• Removed some unused workspace dependencies (running cargo neat)

Are there any user-facing changes?

No

[milenkovicm]

thanks @killzoner
i think changes make sense.
My major comment for discussion if validation should be runtime configuration rather than compile time, I'm leaning towards later, but open for discussion

[killzoner]

thanks @killzoner i think changes make sense. My major comment for discussion if validation should be runtime configuration rather than compile time, I'm leaning towards later, but open for discussion

I pretty much replicated existing config setup, but a compile time config makes sense, as it's a strong default not supposed to be changed frequently.

I will put this specific config under a new feature gate, part of the default features. If you have a name in mind don't hesitate 😄

[milenkovicm]

Im bad with naming things 😁

[killzoner]

Removed all unecessary parametrization, moved config to ShuffleReaderExec. Also duplicated the function for config in FlightService, since i was not sure it made sense to refer to Ballista core config for FlightService config

[milenkovicm]

I have just two minor comments,

[milenkovicm]

thanks @killzoner
i have just a minor comment, i think it makes sense.
will merge this once i kick ballista 51 release process

[killzoner]

thanks @killzoner i have just a minor comment, i think it makes sense. will merge this once i kick ballista 51 release process

Thanks! Updated from your suggestions