Liquid Neural Networks for Real-Time Time-Series Forecasting: Outperforming Transformers on Low-Memory Devices

Vigtige pointer

1. Understand the architecture. Liquid networks use ordinary differential equations to model neuron dynamics continuously.
2. Use fewer parameters. Liquid networks can match transformer accuracy with orders of magnitude fewer weights.
3. Handle irregular sampling. The continuous-time formulation naturally handles variable time steps.
4. Train with standard backprop. Despite the ODE formulation, training uses standard gradient descent.
5. Deploy on edge devices. The small footprint fits on microcontrollers and edge gateways.
6. Measure latency. Liquid networks infer in milliseconds on small devices.
7. Compare against baselines. Benchmark against LSTMs and transformers on your specific task.
8. Tune the time constant. The neuron time constant affects responsiveness; tune per application.

Architecture

Liquid networks model each neuron as a differential equation with a learnable time constant. The network processes inputs continuously, and the state evolves between observations. This allows handling irregular sampling and reduces parameter count.

Production configuration

import torch
import torch.nn as nn

class LiquidCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.tau = nn.Parameter(torch.ones(hidden_size))
        self.W = nn.Linear(input_size, hidden_size)
        self.U = nn.Linear(hidden_size, hidden_size)
    
    def forward(self, x, h, dt):
        tau = torch.abs(self.tau) + 1e-3
        dh = (-h + torch.tanh(self.W(x) + self.U(h))) / tau
        return h + dh * dt

h = torch.zeros(batch, hidden)
for t in range(seq_len):
    h = cell(x[t], h, dt[t])

model = LiquidNet(input_size=4, hidden_size=32, output_size=1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

def train(epoch):
    for x, y, dt in dataloader:
        h = torch.zeros(x.size(0), 32)
        for t in range(x.size(1)):
            h = model.cell(x[:, t], h, dt[:, t])
        pred = model.head(h)
        loss = F.mse_loss(pred, y)
        loss.backward()
        optimizer.step()

python -m torch.onnx.export model input.onnx
onnxruntime-inference input.onnx --input data.npy

Edge cases from production

• Irregular sampling requires accurate timestamps; noisy timestamps degrade performance.
• Very small networks underfit; find the minimum size for your task.
• Continuous-time models are harder to debug than discrete ones; log intermediate states.
• Edge deployment requires quantization; liquid networks quantize well but test accuracy.

Ofte stillede spørgsmål