Creating Frame Processor Plugins

Overview

Frame Processor Plugins are native functions which can be directly called from a JS Frame Processor. (See "Frame Processors")

They receive a frame from the Camera as an input and can return any kind of output. For example, a scanQRCodes function returns an array of detected QR code strings in the frame:

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    const qrCodes = scanQRCodes(frame)
    console.log(`QR Codes in Frame: ${qrCodes}`)
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}

To achieve maximum performance, the scanQRCodes function is written in a native language (e.g. Objective-C), but it will be directly called from the VisionCamera Frame Processor JavaScript-Runtime.

Types

Similar to a TurboModule, the Frame Processor Plugin Registry API automatically manages type conversion from JS <-> native. They are converted into the most efficient data-structures, as seen here:

JS Type	Objective-C/Swift Type	Java/Kotlin Type
number	NSNumber* (double)	Double
boolean	NSNumber* (boolean)	Boolean
string	NSString*	String
[]	NSArray*	ReadableNativeArray
{}	NSDictionary*	ReadableNativeMap
undefined / null	nil	null
(any, any) => void	RCTResponseSenderBlock	(Object, Object) -> void
Frame	Frame*	ImageProxy

Return values

Return values will automatically be converted to JS values, assuming they are representable in the "Types" table. So the following Java Frame Processor Plugin:

@Override
public Object callback(ImageProxy image, Object[] params) {
  return "cat";
}

Returns a string in JS:

export function detectObject(frame: Frame): string {
  'worklet'
  const result = __detectObject(frame)
  console.log(result) // <-- "cat"
}

You can also manipulate the buffer and return it (or a copy of it) by returning a Frame/ImageProxy instance:

@Override
public Object callback(ImageProxy image, Object[] params) {
  ImageProxy resizedImage = new ImageProxy(/* ... */);
  return resizedImage;
}

Which returns a Frame in JS:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  // creates a new `Frame` that's 720x480
  const resizedFrame = resize(frame, 720, 480)

  // by downscaling the frame, the `detectObjects` function runs faster.
  const objects = detectObjects(resizedFrame)
  console.log(objects)
}, [])

Parameters

Frame Processors can also accept parameters, following the same type convention as return values:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, ['qr', 'barcode'])
}, [])

Or with multiple ("variadic") parameters:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const codes = scanCodes(frame, true, 'hello-world', 42)
}, [])

Exceptions

To let the user know that something went wrong you can use Exceptions:

@Override
public Object callback(ImageProxy image, Object[] params) {
  if (params[0] instanceof String) {
    // ...
  } else {
    throw new Exception("First argument has to be a string!");
  }
}

Which will throw a JS-error:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  try {
    const codes = scanCodes(frame, true)
  } catch (e) {
    console.log(`Error: ${e.message}`)
  }
}, [])

What's possible?

You can run any native code you want in a Frame Processor Plugin. Just like in the native iOS and Android Camera APIs, you will receive a frame (CMSampleBuffer on iOS, ImageProxy on Android) which you can use however you want. In other words; everything is possible.

Implementations

Long-running Frame Processors

If your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a copy of the frame and dispatch the actual frame processing to a separate thread.

For example, a realtime video chat application might use WebRTC to send the frames to the server. I/O operations (networking) are asynchronous, and we don't need to wait for the upload to succeed before pushing the next frame, so we copy the frame and perform the upload on another Thread.

@Override
public Object callback(ImageProxy image, Object[] params) {
  String serverURL = (String)params[0];
  ImageProxy imageCopy = new ImageProxy(/* ... */);

  uploaderQueue.runAsync(() -> {
    WebRTC.uploadImage(imageCopy, serverURL);
    imageCopy.close();
  });

  return null;
}

Async Frame Processors with Event Emitters

You might also run some very complex AI algorithms which are not fast enough to smoothly run at 30 FPS (33ms). To not drop any frames you can create a custom "frame queue" which processes the copied frames and calls back into JS via a React event emitter. For this you'll have to create a Native Module that handles the asynchronous native -> JS communication, see "Sending events to JavaScript" (Android) and "Sending events to JavaScript" (iOS).

This might look like this for the user:

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    SomeAI.process(frame) // does not block frame processor, runs async
  }, [])

  useEffect(() => {
    SomeAI.addListener((results) => {
      // gets called asynchronously, goes through the React Event Emitter system
      console.log(`AI results: ${results}`)
    })
  }, [])

  return (
    <Camera frameProcessor={frameProcessor} {...cameraProps} />
  )
}

This way you can handle queueing up the frames yourself and asynchronously call back into JS at some later point in time using event emitters.

Benchmarking Frame Processor Plugins

Your Frame Processor Plugins have to be fast. VisionCamera automatically detects slow Frame Processors and outputs relevant information in the native console (Xcode: Debug Area, Android Studio: Logcat):

🚀 Create your first Frame Processor Plugin for iOS or Android!