[Proposal]: MTP Command-line aliases

[Youssef1313]

Summary

This is a proposal to allow easier migration from VSTest to MTP, by introducing a systematic way of mapping command-line options. The detailed proposal below is more technical, but the less technical summary is that this feature allows us to get back VSTest options such as --logger and --collect, but only when we know that logger or datacollector can be handled in MTP. External owners of VSTest loggers and data collectors can decide to maintain the VSTest syntax, but they definitely will need to implement the extension for MTP. The amount of VSTest extension owners is definitely much less than the whole user space of testing, so this can be manageable.

Background and Motivation

Today, we still have many users on VSTest, and we want to make the migration to MTP as smooth as possible. This proposal will serve the purpose of letting --logger trx, --collect CodeCoverage, etc to remain working, while still producing meaningful errors for unsupported loggers and data collectors.

Proposed Feature

In MTP command-line extensibility, we have the concept of CommandLineOption. Extensions (including test frameworks) can define their own command-line options, and it's an error to define the same command-line option by multiple extensions as it results in an ambiguity.

The proposal here is to have a different concept, CommandLineOptionAlias.

• A command-line option alias, in the initial design, will be forced to be single-arity.
• Multiple extensions can register the same command-line option alias (e.g, --logger`).
• A command-line option alias will have a CanHandle method. Only one of the registered aliases can return true. Otherwise, it's an error.
• If none of the registered alises can handle the given argument, that is also an error.
• It's an error to have a command-line option with the same name as any registered alias.
• An alias will have a Handle method that returns a list of option/value which represents the transformation of that alias.
• Command line aliases are likely needed to be evaluated very early. Hence they won't have access to any services. They will be for the purpose of systematic syntactic transformations.

An example usage for that can be as follows:

testApplicationBuilder.CommandLineAliases.AddProvider(new TrxLoggerProvider());

internal sealed class TrxLoggerProvider : ICommandLineAlias
{
    public override string AliasName => "logger";

    public bool CanHandle(string[] arguments)
    {
        return arguments[0] == "trx" || arguments[0].StartsWith("trx;");
    }

    public IEnumerable<CommandLineAliasTransformation> Handle(string[] arguments)
    {
        if (arguments[0] == "trx") return [new CommandLineAliasTransformation("report-trx", Array.Empty<string>())];

        // Handle cases like trx;LogFileName=...
        // which will map to multiple transformations.
    }

    // Consider: Maybe it's better to combine CanHandle and Handle as a single TryHandle method.
}

• CommandLineParseResult will remain merely the parse result of the command-line. It won't contain the transformations. However, CommandLineManager should consider the transformations. The effect of that is that ICommandLineOptions.IsOptionSet and ICommandLineOptions.TryGetOptionArgumentList will know about the transformations, and is what will set the whole thing to be working correctly together.

Drawbacks

• This will allow doing stuff in two different ways in MTP. Users can be confused why both --logger trx and --report-trx are allowed.
• Extra complexity in MTP extensibility that shouldn't ideally exist, but is existing only to ease migration.
• Before going that way, I would like us to be sure that the change in command-line options is a source of pain for migrations for users. We need to prevent the additional complexity unless it's really really needed.

[Evangelink]

There is currently one possible challenge with the current design of MTP. The -collect option of VSTest accepts many values from (all data collectors) which means that for example TRX and CC extensions would need to share the same option name which is currently forbidden by MTP.

I think it should be easy to workaround that without allowing all extensions to share the same option.

I would also suggest using the obsolete option we have introduced to earn users that the old syntax is obsolete and should be changed.

[Youssef1313]

TRX and CC extensions would need to share the same option name which is currently forbidden by MTP.

That's the whole point of this proposal. Command-line alias is a completely new suggested concept that is different from normal command-line options. Different extensions will be allowed to implement it and implement a "mapping" to original command-line options. It will only be allowed to have a single of those extensions to handle the given value. If multiple extensions claim to support "trx", that will be an error. If none of the extensions claim to support "trx", that will also be an error.

The idea here is that extensions are still required to implement stuff using the MTP way of things, and optionally define mappings for --logger and --collect. It has drawbacks for sure which are stated in the original description.

Side note: TRX is logger (--logger) while CC is a datacollector (--collect).

[Evangelink]

I understood the idea, I think my point was more to say that with MTP we won't be able to have 2 different extensions claiming the same option while it's possible with VSTest (e.g. --collect "ExtensionA;ExtensionB").

Note that I still think it's a good idea to implement this feature.