Re: [RFC PATCH 01/11] target/riscv: Add CLIC CSR mintstatus

[Alistair Francis]

On Tue, Sep 28, 2021 at 6:10 PM Frank Chang <frank.chang@sifive.com> wrote:
>
> On Fri, Jul 2, 2021 at 3:17 PM Alistair Francis <alistair23@gmail.com> wrote:
>>
>> On Fri, Jul 2, 2021 at 4:11 PM LIU Zhiwei <zhiwei_liu@c-sky.com> wrote:
>> >
>> >
>> > On 2021/7/2 下午1:38, Alistair Francis wrote:
>> > > On Thu, Jul 1, 2021 at 6:45 PM Frank Chang <frank.chang@sifive.com> 
>> > > wrote:
>> > >> LIU Zhiwei <zhiwei_liu@c-sky.com> 於 2021年4月20日 週二 上午8:49寫道：
>> > >>>
>> > >>> On 2021/4/20 上午7:23, Alistair Francis wrote:
>> > >>>> On Fri, Apr 9, 2021 at 5:52 PM LIU Zhiwei <zhiwei_liu@c-sky.com> 
>> > >>>> wrote:
>> > >>>>> CSR mintstatus holds the active interrupt level for each supported
>> > >>>>> privilege mode. sintstatus, and user, uintstatus, provide restricted
>> > >>>>> views of mintstatus.
>> > >>>>>
>> > >>>>> Signed-off-by: LIU Zhiwei <zhiwei_liu@c-sky.com>
>> > >>>>> ---
>> > >>>>>    target/riscv/cpu.h      |  2 ++
>> > >>>>>    target/riscv/cpu_bits.h | 11 +++++++++++
>> > >>>>>    target/riscv/csr.c      | 26 ++++++++++++++++++++++++++
>> > >>>>>    3 files changed, 39 insertions(+)
>> > >>>>>
>> > >>>>> diff --git a/target/riscv/cpu.h b/target/riscv/cpu.h
>> > >>>>> index 0a33d387ba..1a44ca62c7 100644
>> > >>>>> --- a/target/riscv/cpu.h
>> > >>>>> +++ b/target/riscv/cpu.h
>> > >>>>> @@ -159,6 +159,7 @@ struct CPURISCVState {
>> > >>>>>        target_ulong mip;
>> > >>>>>
>> > >>>>>        uint32_t miclaim;
>> > >>>>> +    uint32_t mintstatus; /* clic-spec */
>> > >>>>>
>> > >>>>>        target_ulong mie;
>> > >>>>>        target_ulong mideleg;
>> > >>>>> @@ -243,6 +244,7 @@ struct CPURISCVState {
>> > >>>>>
>> > >>>>>        /* Fields from here on are preserved across CPU reset. */
>> > >>>>>        QEMUTimer *timer; /* Internal timer */
>> > >>>>> +    void *clic;       /* clic interrupt controller */
>> > >>>> This should be the CLIC type.
>> > >>> OK.
>> > >>>
>> > >>> Actually there are many versions of CLIC in my branch as different
>> > >>> devices. But it is better to use CLIC type for the upstream version.
>> > >>
>> > >> Hi Alistair and Zhiwei,
>> > >>
>> > >> Replacing void *clic with RISCVCLICState *clic may create a circular 
>> > >> loop
>> > >> because CPURISCVState is also referenced in riscv_clic.h.
>> > >>
>> > >> However, I would like to ask what is the best approach to add
>> > >> the reference of CLIC device in CPURISCVState struct?
>> > >>
>> > >> There may be different kinds of CLIC devices.
>> > >> AFAK, there was another RFC patchset trying to add void *eclic
>> > >> for Nuclei processor into CPURISCVState struct:
>> > >> https://patchwork.kernel.org/project/qemu-devel/patch/20210507081654.11056-2-wangjunqiang@iscas.ac.cn/
>> > >>
>> > >> Is it okay to add the device reference directly into CPURISCVState 
>> > >> struct like that,
>> > >> or we should create some abstraction for these CLIC devices?
>> > >> (However, I'm not sure how big the differences are for these CLIC 
>> > >> devices...)
>> > > I would prefer to not have the CLIC in the struct at all.
>> > >
>> > > Why is the CLIC required from the CPU?
>> >
>> > In my opinion,   the tight coupled interrupt controller, like NVIC in
>> > ARM,  is much different from other devices.
>> > CPU harts need to communicate with it through many ways.
>>
>> Agreed.
>>
>> The difference with RISC-V is that we already have multiple tightly
>> coupled interrupt controllers. We have the PLIC, CLIC, eCLIC and AIA,
>> not to mention possible vendor controllers. Do we really need a CPU
>> struct entry for every single one?
>>
>> I would like to try and keep all of that logic outside of the CPU
>> state. It might not be possible (at least without being a mess) in
>> which case that's fine, but it's at least worth considering.
>>
>> >
>> > We can store the CLIC instance in the struct CPURISCVState or a global
>> > variable.  Is that any other way?
>>
>> We could split the device. So for example the CSRs and other parts
>> that relate to the CPU could be in the CPU while the register mappings
>> and GPIO lines could be it's own device.
>>
>> Another option is to use GPIO lines to indicate the status, but for
>> anything too complex that will be messy.
>
>
> Hi Alistair,
>
> I would like to know whether:
> "Using GPIO lines to indicate the status of CPU"
> is the right approach for future RISC-V CPU/peripherals development?
>
> As RISC-V ecosystem is open to everyone and there are more and more 
> peripherals
> being designed by different vendors.

I think this actually confirms that this is the right direction. If we
are going to have a range of different implementations then we want
them to all use the same basic framework. In this case that would be
GPIO lines.

> I think there is a rising requirement to have an approach for CPU to notify
> those core tightly-coupled peripherals after certain instruction/action has 
> been completed.
> (for this case, CLIC would need to choose the next interrupt to be served 
> after returning from interrupt in mret/sret)
>
> Is it possible to add something like:
>   qemu_irq gpio_out[NUM_OF_GPIOS];
> into CPURISCVState struct so that we can connect the GPIO lines
> at machine/SoC-level, e.g. sifive_u or sifive_e?

Yes, that's entirely possible. The hard part is adding the logic to
the CPU to trigger that line. For standard extensions that is
something we can do, but adding functionality like that for vendor
specific extensions becomes tricky.

>
> When certain instruction or event is executed/completed by CPU.
> It's possible for CPU to notify the peripheral through these GPIO lines,
> and the rest of the tasks can be completed solely by the peripheral itself.
> We don't have to add the hacky codes to the generic CPU routines.
>
> This is just the rough scenario I have come up.
> There might be some other issues I may not think of.
> (For example, how do we determine which GPIO line should be driven by CPU at 
> a certain point?)

Yeah, I agree. Overall I think it's a good concept, and straight
forward to implement. The hard part is doing it in a generic way.

Alistair

>
> Any comments are welcome.
>
> Regards,
> Frank Chang
>
>>
>>
>> >
>> > > I'm guessing we at least need it for CLIC CSR accesses. Could we
>> > > handle that in the CPU and avoid needing a reference to the CLIC?
>> > CSR access is one case. Other cases are:
>> >
>> > 1. When process an interrupt and decide ISP address,  we need to know
>> > current interrupt is vectored or not.
>> >
>> > 2.  When interrupt returned, we need to choose another interrupt to
>> > serve, so that it will not miss any interrupt.
>>
>> Thanks!
>>
>> Alistair
>>
>> >
>> > Thanks,
>> > Zhiwei
>> > >
>> > > Alistair
>> > >
>> > >> Thanks,
>> > >> Frank Chang
>> > >>
>> > >>>
>> > >>>>>    };
>> > >>>>>
>> > >>>>>    OBJECT_DECLARE_TYPE(RISCVCPU, RISCVCPUClass,
>> > >>>>> diff --git a/target/riscv/cpu_bits.h b/target/riscv/cpu_bits.h
>> > >>>>> index caf4599207..c4ce6ec3d9 100644
>> > >>>>> --- a/target/riscv/cpu_bits.h
>> > >>>>> +++ b/target/riscv/cpu_bits.h
>> > >>>>> @@ -165,6 +165,7 @@
>> > >>>>>    #define CSR_MCAUSE          0x342
>> > >>>>>    #define CSR_MTVAL           0x343
>> > >>>>>    #define CSR_MIP             0x344
>> > >>>>> +#define CSR_MINTSTATUS      0x346 /* clic-spec-draft */
>> > >>>>>
>> > >>>>>    /* Legacy Machine Trap Handling (priv v1.9.1) */
>> > >>>>>    #define CSR_MBADADDR        0x343
>> > >>>>> @@ -183,6 +184,7 @@
>> > >>>>>    #define CSR_SCAUSE          0x142
>> > >>>>>    #define CSR_STVAL           0x143
>> > >>>>>    #define CSR_SIP             0x144
>> > >>>>> +#define CSR_SINTSTATUS      0x146 /* clic-spec-draft */
>> > >>>>>
>> > >>>>>    /* Legacy Supervisor Trap Handling (priv v1.9.1) */
>> > >>>>>    #define CSR_SBADADDR        0x143
>> > >>>>> @@ -585,6 +587,15 @@
>> > >>>>>    #define SIP_STIP                           MIP_STIP
>> > >>>>>    #define SIP_SEIP                           MIP_SEIP
>> > >>>>>
>> > >>>>> +/* mintstatus */
>> > >>>>> +#define MINTSTATUS_MIL                     0xff000000 /* mil[7:0] */
>> > >>>>> +#define MINTSTATUS_SIL                     0x0000ff00 /* sil[7:0] */
>> > >>>>> +#define MINTSTATUS_UIL                     0x000000ff /* uil[7:0] */
>> > >>>>> +
>> > >>>>> +/* sintstatus */
>> > >>>>> +#define SINTSTATUS_SIL                     0x0000ff00 /* sil[7:0] */
>> > >>>>> +#define SINTSTATUS_UIL                     0x000000ff /* uil[7:0] */
>> > >>>> The bit fields in the comments are out of date.
>> > >>> I didn't notice it.   Fix it in next version.
>> > >>>
>> > >>> Thanks.
>> > >>>
>> > >>> Zhiwei
>> > >>>
>> > >>>> Alistair
>> > >>>>
>> > >>>>> +
>> > >>>>>    /* MIE masks */
>> > >>>>>    #define MIE_SEIE                           (1 << IRQ_S_EXT)
>> > >>>>>    #define MIE_UEIE                           (1 << IRQ_U_EXT)
>> > >>>>> diff --git a/target/riscv/csr.c b/target/riscv/csr.c
>> > >>>>> index d2585395bf..320b18ab60 100644
>> > >>>>> --- a/target/riscv/csr.c
>> > >>>>> +++ b/target/riscv/csr.c
>> > >>>>> @@ -188,6 +188,12 @@ static int pmp(CPURISCVState *env, int csrno)
>> > >>>>>    {
>> > >>>>>        return -!riscv_feature(env, RISCV_FEATURE_PMP);
>> > >>>>>    }
>> > >>>>> +
>> > >>>>> +static int clic(CPURISCVState *env, int csrno)
>> > >>>>> +{
>> > >>>>> +    return !!env->clic;
>> > >>>>> +}
>> > >>>>> +
>> > >>>>>    #endif
>> > >>>>>
>> > >>>>>    /* User Floating-Point CSRs */
>> > >>>>> @@ -734,6 +740,12 @@ static int rmw_mip(CPURISCVState *env, int 
>> > >>>>> csrno, target_ulong *ret_value,
>> > >>>>>        return 0;
>> > >>>>>    }
>> > >>>>>
>> > >>>>> +static int read_mintstatus(CPURISCVState *env, int csrno, 
>> > >>>>> target_ulong *val)
>> > >>>>> +{
>> > >>>>> +    *val = env->mintstatus;
>> > >>>>> +    return 0;
>> > >>>>> +}
>> > >>>>> +
>> > >>>>>    /* Supervisor Trap Setup */
>> > >>>>>    static int read_sstatus(CPURISCVState *env, int csrno, 
>> > >>>>> target_ulong *val)
>> > >>>>>    {
>> > >>>>> @@ -893,6 +905,13 @@ static int rmw_sip(CPURISCVState *env, int 
>> > >>>>> csrno, target_ulong *ret_value,
>> > >>>>>        return ret;
>> > >>>>>    }
>> > >>>>>
>> > >>>>> +static int read_sintstatus(CPURISCVState *env, int csrno, 
>> > >>>>> target_ulong *val)
>> > >>>>> +{
>> > >>>>> +    target_ulong mask = SINTSTATUS_SIL | SINTSTATUS_UIL;
>> > >>>>> +    *val = env->mintstatus & mask;
>> > >>>>> +    return 0;
>> > >>>>> +}
>> > >>>>> +
>> > >>>>>    /* Supervisor Protection and Translation */
>> > >>>>>    static int read_satp(CPURISCVState *env, int csrno, target_ulong 
>> > >>>>> *val)
>> > >>>>>    {
>> > >>>>> @@ -1644,5 +1663,12 @@ riscv_csr_operations csr_ops[CSR_TABLE_SIZE] 
>> > >>>>> = {
>> > >>>>>        [CSR_MHPMCOUNTER29H] = { "mhpmcounter29h", any32,  read_zero 
>> > >>>>> },
>> > >>>>>        [CSR_MHPMCOUNTER30H] = { "mhpmcounter30h", any32,  read_zero 
>> > >>>>> },
>> > >>>>>        [CSR_MHPMCOUNTER31H] = { "mhpmcounter31h", any32,  read_zero 
>> > >>>>> },
>> > >>>>> +
>> > >>>>> +    /* Machine Mode Core Level Interrupt Controller */
>> > >>>>> +    [CSR_MINTSTATUS] = { "mintstatus", clic,  read_mintstatus },
>> > >>>>> +
>> > >>>>> +    /* Supervisor Mode Core Level Interrupt Controller */
>> > >>>>> +    [CSR_SINTSTATUS] = { "sintstatus", clic,  read_sintstatus },
>> > >>>>> +
>> > >>>>>    #endif /* !CONFIG_USER_ONLY */
>> > >>>>>    };
>> > >>>>> --
>> > >>>>> 2.25.1
>> > >>>>>
>> > >>>>>